Multi-Objective Reinforcement Learning

In multi-objective reinforcement learning (MORL) the agent is provided with multiple feedback signals when performing an action. These signals can be independent, complementary or conflicting. Hence, MORL is the process of learning policies that optimize multiple criteria simultaneously. In this abstract, we briefly describe our extensions to single-objective multi-armed bandits and reinforcement learning (RL) algorithms to make them applicable in multi-objective environments. Bellow, we first summarize the main parts of our multi-objective multi-armed bandits (MOMAB) [2] framework that incorporates techniques from multi-objective optimization. A first straightforward approach is to use a scalarization function which transforms the multi-objective environment into a single-objective environment, and therefore they can be implemented into the standard MAB framework. Since single-objective environments, in general, results in a single optimum, we need to vary the scalarization functions to generate a variety of elements belonging to the Pareto optimal set. The efficiency of these MOMAB algorithms depends very much on the scalarization type. The linear scalarization is the most popular scalarization function due to its simplicity. Moreover, a known problem with linear scalarization is its incapacity to potentially find all the points in a non-convex Pareto set. The Chebyshev scalarization has the advantage that it can find all the points in a non-convex Pareto set at the cost of introducing another parameter i.e. the utopian point, that needs to be optimized. To measure the performance of a single scalarized MOMAB we introduce the scalarized regret that is defined by the difference between the maximum scalarized value and the scalarization of an suboptimal arm. The upper bound of a multiobjective UCB1 that randomly alternates between a number of scalarized UCB1 is then the sum of upper bounds for each single UCB1. While this definition of regret seems natural, it only partially serves our goal because it gathers a collection of independent regrets instead of minimizing the regret of a multi-objective strategy